sfn-glue-sam: Replace deprecated Python shell 3.6 with 3.9#3130
sfn-glue-sam: Replace deprecated Python shell 3.6 with 3.9#3130kakakakakku wants to merge 3 commits into
Conversation
| "--extra-py-files": !Sub "s3://${ArtifactsBucket}/glue/libs/awswrangler-2.12.0-py3-none-any.whl" | ||
| "--library-set": "analytics" | ||
| "--config_path": !Sub "${ETLParametersPath}" | ||
| GlueVersion: "1.0" |
There was a problem hiding this comment.
note: GlueVersion is ignored for Python shell jobs.
You don't need to specify the version of AWS Glue since the parameter --glue-version doesn't apply for AWS Glue shell jobs. Any version specified will be ignored.
https://docs.aws.amazon.com/glue/latest/dg/add-job-python.html
There was a problem hiding this comment.
| 1. Copy glue job script and libs: | ||
| 1. Manually create two Amazon S3 buckets: | ||
| - `ARTIFACTS_BUCKET` — stores the AWS Glue job script. | ||
| - `DATA_BUCKET` — stores the source data for the AWS Glue job. |
There was a problem hiding this comment.
note: These two S3 buckets are not created by the stack. They need to exist before deployment. The original README didn't mention this step at all, so first-time users would likely hit a deploy error without knowing why. Added explicit creation commands with a short note on each bucket's role😀
There was a problem hiding this comment.
Good catch! Can you add cleanup steps for these buckets too?
| Command: | ||
| Name: pythonshell | ||
| PythonVersion: "3" | ||
| PythonVersion: "3.9" |
There was a problem hiding this comment.
There was a problem hiding this comment.
v3.9 is being deprecated. Can you bump this to a more recent version such as 3.13 or 3.14?
ellisms
left a comment
ellisms
left a comment
There was a problem hiding this comment.
Thanks @kakakakakku. I left a few comments to be addressed.
| Command: | ||
| Name: pythonshell | ||
| PythonVersion: "3" | ||
| PythonVersion: "3.9" |
There was a problem hiding this comment.
v3.9 is being deprecated. Can you bump this to a more recent version such as 3.13 or 3.14?
| 1. Copy glue job script and libs: | ||
| 1. Manually create two Amazon S3 buckets: | ||
| - `ARTIFACTS_BUCKET` — stores the AWS Glue job script. | ||
| - `DATA_BUCKET` — stores the source data for the AWS Glue job. |
There was a problem hiding this comment.
Good catch! Can you add cleanup steps for these buckets too?
| ## Cleanup | ||
| 1. Delete the stack: | ||
| ```bash | ||
| > aws cloudformation delete-stack --stack-name STACK_NAME |
There was a problem hiding this comment.
we should use sam delete since this is deployed via SAM
3 participants
Issue #, if available:
N/A
Description of changes:
Hi😀 Thanks for the useful patterns!
This pattern was last updated about 5 years ago and currently doesn't work because Python shell 3.6 reached end of support on March 1, 2026. I made the following updates to bring it back to a working state.
Check
sam deploycompleted successfully and works good.$ aws s3 mb s3://kakakakakku-sfn-glue-sam-artifacts make_bucket: kakakakakku-sfn-glue-sam-artifacts $ aws s3 mb s3://kakakakakku-sfn-glue-sam-data make_bucket: kakakakakku-sfn-glue-sam-data $ aws s3 cp code/glue s3://kakakakakku-sfn-glue-sam-artifacts/glue --recursive $ sam build --template code/cloudformation/stack.yaml $ sam deploy --guided CloudFormation outputs from deployed stack ----------------------------------------------------------------------------------------------------------------------------- Outputs ----------------------------------------------------------------------------------------------------------------------------- Key GlueJobName Description - Value Feature Engineering Key ETLStepFunctionsARN Description - Value arn:aws:states:ap-northeast-1:000000000000:stateMachine:ETLStepFunctions-9pMndnlymAiz ----------------------------------------------------------------------------------------------------------------------------- Successfully created/updated stack - sfn-glue-sam in ap-northeast-1Thank you😀
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.